Simulating Classifier Outputs for Evaluating Parallel Combination Methods

The use of artificial outputs generated by a classifier simulator has recently emerged as a new trend to provide an underlying evaluation of classifier combination methods. In this paper, we propose a new method for the artificial generation of classifier outputs based on additional parameters which provide sufficient diversity to simulate, for a problem of any number of classes and any type of output, any classifier performance. This is achieved through a two-step algorithm which first builds a confusion matrix according to desired behaviour and secondly generates, from this confusion matrix, outputs of any specified type. We provide the detailed algorithms and constraints to respect for the construction of the matrix and the generation of outputs. We illustrate on a small example the usefulness of the classifier simulator.